Blog/kagent security lab - #264

Hey @Prianshu-git, thanks for this one. I went through it properly, checked the kagent API claims against the source, and pulled up all five images. Short version: the technical substance is solid and this is close, closer than most first drafts we get. There's a handful of things to fix before it goes out, and two of them would actually block a reader from following along.

First, credit where it's due, because I verified this rather than assuming it. requireApproval is a real field on McpServerTool in kagent.dev/v1alpha2, nested exactly where you put it (sibling of toolNames, under mcpServer), and kagent enforces a CEL rule that every entry in requireApproval must also appear in toolNames. Your YAML satisfies that. The ModelConfig shape checks out too (model, provider, ollama.host are all real fields), and chart version 0.9.12 exists. That matters a lot to me, because the last blog I reviewed had a centerpiece YAML field that didn't exist in the project at all. Yours holds up.

The lab also reads like you genuinely ran it. The stalled ollama pull progress line sitting at 94% with 16 KB/s and a 58m29s ETA is not something anyone invents, and neither is "I switched models during the lab". Keep that texture, it's the best thing about the post.

Two things that block a reader

There's no way to open the dashboard. Step 6 says "Open the kagent dashboard at http://localhost:8080", but nothing anywhere in the post starts a port-forward. A reader following along hits that line and stops. kagent's own CLI prints this when you run kagent dashboard:

kubectl port-forward -n kagent service/kagent-ui 8082:8080

Note the port: 8080 is the in-cluster service port, and the local one is 8082, so http://localhost:8080 is also wrong as written. Please add the port-forward as its own step before Step 6 and fix the URL.

The lab files are never obtained. Step 1 opens with kind create cluster --config 00-cluster/kind-config.yaml, and later steps reference 01-local-llm/ollama-deployment.yaml, 01-local-llm/modelconfig.yaml, 02-first-agent/agent.yaml and 03-human-in-the-loop/hitl-agent.yaml. None of those exist on the reader's machine, and the repo that has them is linked in the very last line of the post. Please add a clone step in the prerequisites:

git clone https://github.com/Prianshu-git/Kagent-demo
cd Kagent-demo

I'd also inline kind-config.yaml since it's the very first command and it's presumably short. Everything else can stay as a repo reference.

The images, which is where the most upside is

You committed five images but only reference three. 1.png and 3.png are sitting in the PR unused, and 3.png in particular is the single best screenshot in the set. Let me explain why.

3.png shows local-k8s-agent being asked "how many namespaces currently in my cluster". It calls k8s_get_available_api_resources and answers "There are currently 51 namespaces in your cluster." That answer is wrong twice over: a kind cluster with kagent and Ollama on it has something like seven namespaces, and the tool it reached for lists API resource types, not namespaces. The model grabbed the wrong tool and then reported that tool's item count as a namespace count.

That is a perfect, real illustration of the exact point you make in "Notes on model selection and behavior", and right now that section has no evidence behind it. Put 3.png there. It's much more convincing than the prose alone.

It also complicates a claim you make a few lines later, and I think honestly makes the post better: you write that k8s_get_resources "doesn't hallucinate. If the agent uses it, the answer is real." Your own screenshot shows the failure mode is upstream of that, the model picking the wrong tool in the first place. Worth softening to something like: the tools return ground truth, but nothing guarantees the model calls the right one, which is exactly why read-only scoping and approval gates matter.

Two more image notes:

• 2.png is the wrong agent for where it sits. It's placed in Step 6 under the read-only agent with alt text "local-K8s-agent", but the screenshot clearly shows kagent/local-hitl-agent answering. 3.png is the one that actually shows local-k8s-agent. Simplest fix: use 3.png's sibling framing for Step 6, or just re-take the read-only screenshot.
• Steps 7 and 8 have no screenshots at all, and they're the whole point of the post. The approval gate pausing in the UI, and you clicking approve, then reject, is the money shot of this entire lab. 1.png is close but shows a read response rather than a pending approval. If you can, grab two: one of the pending-approval state, one after rejection. That single addition would lift the post more than any prose change.

On file names and weight: 1.png, 2.png, 3.png, cover1.png should be descriptive (hitl-approval-pending.png and so on), and the two big ones need compressing. cover1.png is 1.9MB and Architecture.png is 1.2MB. Aim for under ~300KB each, WebP is fine.

On the cover image specifically: it's clearly AI-generated, and it shows. The laptop UI in it has garbled nonsense words down the left sidebar where menu labels should be. That's the classic tell, and readers spot it instantly on a post whose whole subject is trusting AI systems. My honest suggestion: drop it and use a clean screenshot of the actual approval gate as the cover. A real UI showing "Pending approval" with Approve and Reject buttons is a stronger, more credible image for this post than any generated art. If you'd rather keep generated art, at least regenerate it without legible-but-fake text.

The architecture diagram, on the other hand, is genuinely good. Clear, accurate to the lab, nicely on-brand. One nit: it shows an "RBAC-scoped" badge, but Part 1 doesn't cover RBAC at all (you've scoped that to Part 2), so either drop that badge or note it's coming next.

Accuracy details

Your performance numbers contradict each other three ways. Step 4 says qwen2.5:1.5b is "roughly 5x faster than larger alternatives". Later the same section says "roughly 5-8x faster". And the concrete figures given, 2 to 4 tokens/sec versus 0.25 tokens/sec, work out to 8x to 16x. Pick the number your measurements actually support and use it consistently. I'd also double-check the 0.25 t/s figure for llama3.2:3b, since that's unusually slow even for CPU-only inference and it's the weakest-looking claim in the post. If it's real, say what the CPU and memory limits on the pod were, and it becomes credible instead of surprising.

Show where the timing logs come from. You paste slot print_timing lines but never show the command that produced them. Add the kubectl logs -n ollama deploy/ollama line above that block so readers can reproduce it.

The "Current cluster status" output doesn't match a fresh lab. The pod ages read 13h, 14h and 17h, which tells the reader this came from your long-running dev cluster rather than the walkthrough they just did. Someone following along will see ages in minutes. Either re-capture it from a clean run or add a line acknowledging it's from a cluster that had been up a while.

Step 2's wait may fire too early. kubectl wait --for=condition=ready pod --all -n kagent immediately after helm install can fail with "no matching resources" because the pods don't exist yet. Worth either adding a short sleep, or waiting on the deployment instead.

Repo hygiene

This PR adds redirect rules for a post that doesn't exist. Your regenerated vercel.json and public/_redirects include entries for zero-trust-istio-sidecar-vs-ambient, which isn't in this branch and isn't in main. That looks like you ran the generator while another post was sitting in your working tree. Merging as-is would publish a rewrite rule pointing at a page that 404s, and we've been bitten by exactly that class of bug before (it caused a refresh loop on the site). Please regenerate the redirects with only your post present:

git checkout origin/main -- vercel.json public/_redirects public/_worker.js
node scripts/generate-redirects.mjs

The filename and slug don't match. The file is kagent-security-lab.md but the frontmatter slug is kagent-part-1-local-ai-agent-kubernetes. Our guide asks for these to be identical, otherwise the automatic "Edit this page on GitHub" link 404s. The slug is the better name, so rename the file to kagent-part-1-local-ai-agent-kubernetes.md (and the image folder can stay as-is or move to match, your call).

You're not in content/authors.json. The frontmatter says author: prianshu-mukherjee but there's no such key, so the post would publish under the generic "Kubesimplify" byline instead of yours. Add yourself, per Step 3 of WRITING-A-BLOG.md, with an avatar in public/img/authors/.

Commits. There's a merge commit in the branch, and the DCO sign-off is on only one of the two commits. Squash into a single signed-off commit:

git reset --soft origin/main
git commit -s -m "Add blog post: kagent Part 1, local Kubernetes-native AI agent with human-in-the-loop approval"
git push --force

Style and copy

• Two em dashes need replacing (we don't use them): the seoDescription ("with Ollama — read-only") and the prerequisites line ("Keep the model small — this walkthrough"). There's also an en dash in "45–60 minutes" on the reading-time line.
• Drop the reading-time line entirely. The site calculates and displays reading time automatically from the content, so "Reading time: ~18 minutes" will contradict it. Keep the hands-on estimate if you like, just move it into the intro prose.
• Seven tags, we ask for two to five. I'd cut to ["kagent", "kubernetes", "ai-agents", "human-in-the-loop"].
• Typos and small grammar: "explain situtation" (missing word and typo), "That's no a bug" should be "not", "posture don't grant a tool" needs a colon or dash, "proceeding on guesses make things even worse" (subject-verb), "this lab demonstrated one thing a local AI agent can operate" needs a colon after "one thing", "The architecture is explicit :" and "qwen2.5:1.5b ." have stray spaces before punctuation, and "llama 3.2:3b" is written three different ways across the post (pick llama3.2:3b).
• One structural repeat: the three-bullet list right after the intro ("An AI chatbot can explain... A kagent agent can decide what to inspect next...") restates the opening paragraph almost word for word. Cut one of them.

Where it lands

The bones are good and the technical claims hold, which is the hard part. What it needs is the two reader-blocking gaps closed (port-forward, clone step), the unused screenshots put to work (especially 3.png in the hallucination section), a real screenshot of the approval gate in Steps 7 and 8, the redirect regeneration, and the copy pass.

Do that and I'm glad to publish it. Ping me when you've pushed and I'll spin up a preview and take another look.

This is a really solid turnaround @Prianshu-git. I went through the whole thing again and every blocker from last round is closed.

Confirmed fixed: the file is renamed to match the slug, Step 6 port-forwards to the right port, the clone step is in the prerequisites, the reading-time line is gone, all 11 images are descriptively named with zero orphans and zero broken links, and the performance numbers are now internally consistent instead of contradicting themselves three ways. I also regenerated the redirect files from scratch on your branch and got an empty diff, so the generated files are correct and the stray zero-trust entries from last time are gone. You're already in authors.json from the Istio post, so your byline resolves.

Two things you did better than I asked for. Using the pending-approval screenshot as the cover is a much stronger choice than the generated art, since it shows the actual thing the post is about. And you used the hallucination screenshot exactly right: calling out that the model grabbed k8s_get_available_api_resources and reported that tool's item count as a namespace count, rather than inventing a number, is a sharper observation than "small models hallucinate." Softening the "it doesn't hallucinate" line to point at tool selection as the real risk surface is the correct nuance, and it makes the least-privilege argument land harder.

Adding the pod limits (2 vCPU / 4Gi) was the detail I most wanted, because it explains the timings. Which brings me to the one substantive thing I'd still change.

The one thing I'd still change: the performance framing

The numbers are real, I'm not questioning them. My problem is with what the post concludes from them, and there's a specific measurement that would settle it.

An 8x to 16x gap between a 1.5B and a 3B model is too large to be a parameter-count story. Twice the parameters means roughly twice the compute per token, so you'd expect something near 2x on the same cores. 8x to 16x says something else was going on, and the post currently attributes it to capability ("slightly lower reasoning capability", "if you need deeper reasoning at the cost of latency, switch back to llama3.2:3b"). That framing tells readers the slow model was thinking harder, when the likelier story is that it was starved.

The most probable culprit is right there in your own output: you have two ModelConfigs, default-model-config on llama3.2:3b and local-model-config on qwen2.5:1.5b, and your ollama list shows llama3.2:latest and llama3.2:3b too. If more than one model was resident at once, that's 3GB+ of weights alone inside a 4Gi limit, before KV cache and runtime overhead. Your own troubleshooting section even mentions the Ollama pod hitting memory pressure. The other candidate is simply the 2 vCPU ceiling.

I don't want to hand you my guess as fact, so here's how to find out. This takes about ten minutes.

1. See what's actually loaded. This is the diagnostic that matters most, and it also tells you whether inference ran on CPU or something else:

kubectl exec -n ollama deploy/ollama -- ollama ps

If that lists more than one model, you've found your answer. SIZE is the resident footprint and PROCESSOR confirms CPU.

2. Check whether the pod was being squeezed:

kubectl top pod -n ollama
kubectl get pod -n ollama -l app=ollama \
  -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}{"\n"}'
kubectl describe pod -n ollama -l app=ollama | grep -A3 "Last State"

A non-zero restart count or an OOMKilled last state proves memory pressure outright.

3. Benchmark each model cleanly, one resident at a time. This sidesteps the agent loop entirely, so you're measuring the model rather than the whole system:

kubectl exec -n ollama deploy/ollama -- ollama stop llama3.2:3b
kubectl exec -n ollama deploy/ollama -- ollama run --verbose qwen2.5:1.5b \
  "Write exactly 100 words about Kubernetes scheduling."

kubectl exec -n ollama deploy/ollama -- ollama stop qwen2.5:1.5b
kubectl exec -n ollama deploy/ollama -- ollama run --verbose llama3.2:3b \
  "Write exactly 100 words about Kubernetes scheduling."

--verbose prints prompt eval rate and eval rate in tokens/sec. Same prompt, one model in memory at a time, so it's an apples-to-apples number you can publish with confidence.

4. Then raise the limits and run step 3 again:

kubectl set resources deployment/ollama -n ollama \
  --limits=cpu=6,memory=8Gi --requests=cpu=2,memory=4Gi
kubectl rollout status deployment/ollama -n ollama

Whatever comes back, you have a better post. If the gap collapses to roughly 2x once only one model is resident, you've got a genuinely interesting finding: on a memory-capped node the model that fits wins regardless of which one is smarter, which is a far more useful lesson than "bigger is slower." If the gap holds at 8x, then you've verified your original claim properly and can state it with the measurement behind it.

For the two places the post makes the claim, here's the shape I'd use once you have numbers. Step 4:

Why qwen2.5:1.5b? It's less than half the size of llama3.2:3b (986 MB vs. 2.0 GB), which matters more than raw parameter count on a memory-capped node. Measured with ollama run --verbose inside the pod, one model resident at a time, at limits of N vCPU / _N_Gi: qwen2.5:1.5b generated N tokens/sec versus N tokens/sec for llama3.2:3b. For structured tool calling, which is what agents actually need, the smaller model is more than adequate.

And in the performance section, swap the "honest tradeoff of local CPU inference" line for something that separates the two variables, along the lines of: these numbers come from a pod capped at N vCPU / _N_Gi, raising those limits changes them substantially, and the floor you can't tune away is that each interaction costs several full model passes.

Also worth adding: tell readers what wait to expect. "You probably noticed each interaction took a while" is honest but not actionable. Your own screenshots have the numbers: the read-only pod-listing answer used 1,183 output tokens, so at 2 to 4 tokens/sec that single answer took five to ten minutes. The approval interaction was lighter at 144 tokens, about a minute. Across Step 7 and the four parts of Step 9 that's roughly 15 to 40 minutes of waiting on top of the installs and the model pull. Put that in the prerequisites and nobody wonders whether their cluster has hung.

And name the host, since none of the timings are reproducible without it:

system_profiler SPHardwareDataType | grep -E "Chip|Total Number of Cores|Memory:"
docker info --format 'Docker: {{.NCPU}} CPUs, {{.MemTotal}} bytes'

One line with the chip, core count, RAM, and Docker's allocation is all it needs.

To be clear about where I land: none of this undermines the lab. What you're teaching (the approval boundary, least-privilege tool scoping, the CRD shape) behaves identically at 2 tokens/sec and 200 tokens/sec, you just wait longer to watch it work. The post is right that fully local is workable and not low-latency. I'd just rather it pin the latency on limits the reader can change than on CPU inference in general, because the first is a knob and the second sounds like a dead end.

Three small things

• architecture-diagram.png is 1.5MB, now the only oversized asset in the post. Everything else is comfortably under 150KB. Worth getting it under ~300KB, and since it's a flat-colour diagram it should compress hard (WebP or SVG both work well here).
• The publish date is still 2026-08-18. If we merge this week it'll land backdated and sort below newer posts. Bump it to the day we publish.
• Your DCO sign-off uses prianshu@prianshus-MacBook-Air.local, which is your local machine hostname rather than a real address. Some DCO checks reject that. Setting git config user.email to your GitHub address and re-signing (git commit --amend -s --no-edit) sorts it.

Fix the performance framing and compress that diagram and I'm happy to ship this. The lab itself is in good shape now, and the approval-gate walkthrough with real screenshots is the strongest part.